[Feat] Add Ray Cron Job #4159

machichima · 2025-10-30T14:12:53Z

Why are these changes needed?

Support cron job scheduling. Following this design docs and implement milestone 1 in this PR

Main changes:

Add RayCronJob CRD and controller
Add feature gate for enabling RayCronJob
Add unit test

Test

Apply the sample YAML ray-operator/config/samples/ray-cronjob.sample.yaml. RayJobs are being scheduled every minute:

Trigger validation error

Related issue number

Following comment: #2426 (comment)

Closes #2426

Checks

I've made sure the tests are passing.
Testing Strategy
- Unit tests
- Manual tests
- This PR is not tested :(

…-poc

ray-operator/apis/ray/v1/raycronjob_types.go

ray-operator/controllers/ray/raycronjob_controller.go

ray-operator/apis/ray/v1/raycronjob_types.go

win5923 · 2025-11-26T17:07:51Z

ray-operator/apis/ray/v1/raycronjob_types.go

+type RayCronJobSpec struct {
+	// JobTemplate defines the job spec that will be created by cron scheduling
+	JobTemplate *RayJobSpec `json:"jobTemplate"`
+	// Schedule is the cron schedule string
+	Schedule string `json:"schedule"`
+}


Not sure if this is alright, but I think we should introduce a separate struct (e.g. RayJobTemplateSpec) to hold both the metadata and the spec for the generated RayJob, similar to how Kubernetes models JobTemplateSpec in CronJob.

This would allow users to specify metadata inside jobTemplate, which we can then propagate to the created RayJob. It also keeps the API aligned with common Kubernetes patterns.

WDYT?

Reference:
https://github.com/kubernetes/kubernetes/blob/af9fb799ef09bbdb0b2b40b4e441f2ffccaffe18/pkg/apis/batch/types.go#L94-L105

Not sure if we want this. In RayJobSpec when we specify the RayClusterSpec, we only set RayClusterSpec itself rather than also letting user set the ObjectMeta.

kuberay/ray-operator/apis/ray/v1/rayjob_types.go

Line 206 in 5a836b6

RayClusterSpec *RayClusterSpec `json:"rayClusterSpec,omitempty"`

cc @rueian for confirmation.

No need for it at the moment, this can be a future work.

The current structure looks good to me already.

Signed-off-by: machichima <nary12321@gmail.com>

Future-Outlier · 2025-11-29T01:24:23Z

ray-operator/config/samples/ray-cronjob.sample.yaml

+      rayVersion: '2.46.0' # should match the Ray version in the image of the containers
+      # Ray head pod template


can we use 2.52.0 version? thank you!

Updated in 6e3c37a. Thank you!

Signed-off-by: machichima <nary12321@gmail.com>

win5923 · 2025-12-01T15:55:01Z

ray-operator/controllers/ray/raycronjob_controller.go

+		// This is the only 2 places where we update the RayCronJob status. This will directly
+		// update the ScheduleStatus to ValidationFailed if there's validation error
+		if err = r.updateRayCronJobStatus(ctx, originalRayCronJobInstance, rayCronJobInstance); err != nil {
+			logger.Info("Failed to update RayCronJob status", "error", err)
+			return ctrl.Result{RequeueAfter: RayCronJobDefaultRequeueDuration}, err
+		}


No need to update status for validation failed since we not use ScheduleStatus for RayCronJob.

win5923 · 2025-12-01T16:33:04Z

ray-operator/controllers/ray/raycronjob_controller_unit_test.go

+	// Create RayCronJob with invalid cron schedule
+	rayCronJob := &rayv1.RayCronJob{
+		ObjectMeta: metav1.ObjectMeta{
+			Name:      "invalid-cronjob",
+			Namespace: "default",
+		},
+		Spec: rayv1.RayCronJobSpec{
+			Schedule: "invalid cron string",
+			JobTemplate: &rayv1.RayJobSpec{
+				Entrypoint: "python test.py",
+				RayClusterSpec: &rayv1.RayClusterSpec{
+					HeadGroupSpec: rayv1.HeadGroupSpec{
+						Template: corev1.PodTemplateSpec{
+							Spec: corev1.PodSpec{
+								Containers: []corev1.Container{
+									{
+										Name:  "ray-head",
+										Image: "rayproject/ray:2.9.0",
+									},
+								},
+							},
+						},
+					},
+				},
+			},
+		},
+	}


Can we create a template function for the ut? like this:

kuberay/ray-operator/controllers/ray/raycluster_controller_test.go

Line 45 in 8aa4d10

func rayClusterTemplate(name string, namespace string) *rayv1.RayCluster {

win5923

LGTM, Really cool!

rueian · 2025-12-01T20:28:59Z

ray-operator/controllers/ray/raycronjob_controller.go

+
+	if rayCronJobInstance.Status.LastScheduleTime == nil {
+		// The new RayCronJob, not yet scheduled
+		rayCronJobInstance.Status.LastScheduleTime = &metav1.Time{Time: now}


Why do we only update the LastScheduleTime without creating the rayjob in the first reconciliation?

The first reconciliation is just to prepare the schedule. We don't create the job yet because we need to check if the scheduled time has actually arrived (e.g., if the job runs every minute but it's currently 15:10:30, we shouldn't create it yet)

Therefore, the first reconcile just sets the current time as LastScheduleTime and sets the requeue delay to future schedule time - now."

I have the same question, is this how kuberentes job api works?

I have the same question, is this how kuberentes job api works?

Not really, Kubernetes CronJob uses the CreationTimestamp.Time to compute the next schedule time when evaluating it for the first time:

func mostRecentScheduleTime(cj *batchv1.CronJob, now time.Time, ...) { earliestTime := cj.ObjectMeta.CreationTimestamp.Time if cj.Status.LastScheduleTime != nil { earliestTime = cj.Status.LastScheduleTime.Time } // ... }

https://github.com/kubernetes/kubernetes/blob/ec1bf8a4f3a5f054065225dc8275c66b93310d17/pkg/controller/cronjob/utils.go#L94-L123

The CronJob also use two schedule times (t1 and t2) calculate the next time to handle missed schedules, if the current time falls between t1 and t2, the controller will create a Job.

t1 := schedule.Next(earliestTime) t2 := schedule.Next(t1) if now.Before(t1) { return earliestTime, nil, missedSchedules, nil } if now.Before(t2) { return earliestTime, &t1, missedSchedules, nil }

https://github.com/kubernetes/kubernetes/blob/ec1bf8a4f3a5f054065225dc8275c66b93310d17/pkg/controller/cronjob/utils.go#L115-L180

sequenceDiagram participant R as Reconciler participant M as mostRecentScheduleTime() participant K as K8s API Note over R: CronJob created at 10:05<br/>Schedule: "0 * * * *" R->>M: First reconcile (10:05) M->>M: earliestTime = CreationTimestamp (10:05) M->>M: t1 = schedule.Next(10:05) = 11:00 M->>M: now (10:05) < t1 (11:00) ? M-->>R: Return nil (not time yet) Note over R: LastScheduleTime stays nil<br/>Requeue until 11:00 R->>M: Second reconcile (11:00) M->>M: earliestTime = CreationTimestamp (10:05) M->>M: t1 = 11:00, t2 = 12:00 M->>M: now (11:00) >= t1 && now < t2 ? M-->>R: Return &t1 (11:00) ✅ R->>K: CreateJob(scheduledTime: 11:00) K-->>R: Job created ✅ R->>K: Update Status.LastScheduleTime = 11:00 Note over R: First Job created!

Loading

And the LastScheduleTime is only updated after creating a Job.

scheduledTime, err := nextScheduleTime(logger, cronJob, now, sched, jm.recorder) .... cronJob.Status.Active = append(cronJob.Status.Active, *jobRef) cronJob.Status.LastScheduleTime = &metav1.Time{Time: *scheduledTime} updateStatus = true

https://github.com/kubernetes/kubernetes/blob/ec1bf8a4f3a5f054065225dc8275c66b93310d17/pkg/controller/cronjob/cronjob_controllerv2.go#L668-L670

If i was wrong, please correct me.

Future-Outlier · 2025-12-02T14:25:20Z

ray-operator/controllers/ray/raycronjob_controller.go

+
+	if rayCronJobInstance.Status.LastScheduleTime == nil {
+		// The new RayCronJob, not yet scheduled
+		rayCronJobInstance.Status.LastScheduleTime = &metav1.Time{Time: now}


I have the same question, is this how kuberentes job api works?

Future-Outlier · 2025-12-02T14:26:42Z

ray-operator/controllers/ray/raycronjob_controller.go

+func (r *RayCronJobReconciler) updateRayCronJobStatus(ctx context.Context, oldRayCronJob *rayv1.RayCronJob, newRayCronJob *rayv1.RayCronJob) error {
+	logger := ctrl.LoggerFrom(ctx)
+	oldRayCronJobStatus := oldRayCronJob.Status
+	newRayCronJobStatus := newRayCronJob.Status
+	if oldRayCronJobStatus.LastScheduleTime != newRayCronJobStatus.LastScheduleTime {
+
+		logger.Info("updateRayCronJobStatus", "old RayCronJobStatus", oldRayCronJobStatus, "new RayCronJobStatus", newRayCronJobStatus)
+		if err := r.Status().Update(ctx, newRayCronJob); err != nil {
+			return err
+		}
+	}
+	return nil
+}


Does this include the first time we initialize LastScheduleTime?

machichima added 17 commits October 30, 2025 22:10

feat: init RayCronJob controller and add CRs

384c0e1

feat: update status

027ccaf

feat: add validate

e99572d

feat: add update status function

3b4a2cb

refactor: udpate LastScheduleTime type

42e6ec2

feat: implement logic for diff ScheduleStatus

f3c5faa

build: regen CRD

e7c36c1

feat: correctly create reconciler and add loggings

93916ec

feat: check if it's time for schedule new rayjob

c2fe872

feat: add raycronjob example yaml

9cc5ecc

Merge branch 'master' of github.com:ray-project/kuberay into cron-job…

11d3cd5

…-poc

fix: remove StatusScheduled

b104f61

build: make sync

eade1ee

refactor: move validate to validation.go

8568e66

test: add validation and raycronjob unit test

ff272e0

feat: add feature gate

aa91293

test: update test

4c053d0

machichima changed the title ~~[POC] Add Ray Cron Job~~ [Feat] Add Ray Cron Job Nov 24, 2025

fix: update helm chart rules

3f04c07

machichima marked this pull request as ready for review November 24, 2025 13:03

machichima requested review from MortalHappiness, andrewsykim, kevin85421 and rueian as code owners November 24, 2025 13:03

CheyuWu self-requested a review November 24, 2025 17:01

win5923 reviewed Nov 24, 2025

View reviewed changes

ray-operator/apis/ray/v1/raycronjob_types.go Show resolved Hide resolved

win5923 reviewed Nov 24, 2025

View reviewed changes

ray-operator/controllers/ray/raycronjob_controller.go Outdated Show resolved Hide resolved

fix: add OwnerReference from cronjob to rayjob

584abd3

win5923 reviewed Nov 26, 2025

View reviewed changes

ray-operator/apis/ray/v1/raycronjob_types.go Outdated Show resolved Hide resolved

win5923 reviewed Nov 26, 2025

View reviewed changes

ray-operator/apis/ray/v1/raycronjob_types.go Outdated Show resolved Hide resolved

win5923 reviewed Nov 26, 2025

View reviewed changes

machichima added 3 commits November 28, 2025 20:45

fix: field order for RayCronJob struct

c8c93ac

Signed-off-by: machichima <nary12321@gmail.com>

fix: remove schedule status

970c3ab

Signed-off-by: machichima <nary12321@gmail.com>

build: make generate

a38d13a

Signed-off-by: machichima <nary12321@gmail.com>

Future-Outlier reviewed Nov 29, 2025

View reviewed changes

fix: update example yaml to use ray 2.52.0 image

6e3c37a

Signed-off-by: machichima <nary12321@gmail.com>

win5923 reviewed Dec 1, 2025

View reviewed changes

win5923 approved these changes Dec 1, 2025

View reviewed changes

rueian reviewed Dec 1, 2025

View reviewed changes

Future-Outlier reviewed Dec 2, 2025

View reviewed changes

		rayVersion: '2.46.0' # should match the Ray version in the image of the containers
		# Ray head pod template

[Feat] Add Ray Cron Job #4159

Are you sure you want to change the base?

[Feat] Add Ray Cron Job #4159

Uh oh!

Conversation

machichima commented Oct 30, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Why are these changes needed?

Test

Related issue number

Checks

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

win5923 Nov 26, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

win5923 Dec 1, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

win5923 Dec 1, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

win5923 left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

machichima commented Oct 30, 2025 •

edited

Loading

win5923 Nov 26, 2025 •

edited

Loading

win5923 Dec 1, 2025 •

edited

Loading

win5923 Dec 1, 2025 •

edited

Loading